In conventional software testing systems, when changes are made to source code that is to be tested, software engineers manually determine which tests to run on the source code and in which environments to run those tests in order to prove the quality of the changes to the source code. This process can be error prone because an engineer may not understand the various factors that may affect performance, correctness, security, reliability, and/or other aspect(s) of the source code. Accordingly, tests that are capable of testing such factors may not be identified by the software engineers to be run on the source code.
A variety of systems have been developed for automating testing of software, though each system has its limitations. For instance, some of today's state of the art systems, such as Visual Studio 2010®, enable developers and testers to use a feature that is commonly referred to as “Test Impact Analysis” to relate source code that is changed to a set of tests that exercise the changed source code (and then run those tests). Information about Test Impact Analysis may be found at http://visualstudiomagazine dot com/articles/2011/02/10/test-impact-analysis dot aspx, for example. Systems that offer Test Impact Analysis typically record the source code that is exercised by each test and create a mapping between the source code and the tests. With such a system, an engineer can determine an appropriate subset of tests to run on the source code based on runtime dependencies between components of the source code. However, this approach merely accounts for dependencies between the components.